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ABSTRACT 
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multiple comparisons (post hoc rr unplanned versui. a priori or 
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Textbook views and recomrr.endations regarding the use of these two 
types of multiple comparisons are summarized. Analysis of variance 
(ANOVA) and related methods remain popular, although other methods 
are sometiires more useful and although the frequency of use of these 
methods in published research is declining. Researchers conducting an 
ANOVA fr»5quently conduct unplanned or post hoc tests to determine 
exactly whare statistically significant group differences actually 
ariee. Various methodologists have referred to this practice as 
"fishing", "milking", or "data snooping". It is widely known that 
planned multiple comparisons have greater statistical power than do 
unplanned multiple comparisons, '^econdly, and more importantly, 
planned comparisons tend to make the researcher think in advance. 
Greiuate study in stati.stics snould reflect the importance of planned 
versus post hoc statistical design. In addition, there is a need for 
further investigation of current statistics textbooks being used in 
the different schools ox education, business, and the behavioral 
sciences to determine if the issue of planned versus post hoc tests 
is being adequately addressed : instructional materials. A 63-item 
list of references, 4 data tablu . , and a list of srilected computer 
control cards are included. (TJH) 
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Abstract 

ANOVA and related methods remain popular, though other methods are 
sometimes more useful (Thompson, 1986) and though the frequtncy of use of 
these methods in published research is declining (Elmore & Woehlke, 1988; 
Gaither & Glorfeld, 1985). Researchers conducting ANOVA frequently 
conduct unplanned or post hoc tests to determine exactly where 
statistically significant group differences actually arista. Various 
methodologists have referred to this practice as "fishing" (Minium & 
Clarke, 1982, p. 321), "milking" (Keppel, 1982, p. 150) or "data snooping" 
(Kirk, 1969, p. 73; 1984, p. 360; Pedhazur, 1982, p. 305). An alternative 
is to conduct planned or a priori tests rather than post hoc or unplanned 
tests. It is widely known that planned multiple comparisons have greater 
power than unplanned multiple comparisons (Thompson, 1985, 1988b, 1988c, 
1989). The purpose of this paper is to review and summarize textbook 
views and recommendations regarding the use of these two types of multiple 
comparisons (Thompson, 1987). Such a review may be useful in 
understanding the origins of several features of contemporary analytic 
practice. 



Researchers have several analytical methods for evaluating 
differences between means. Often each method will provide different 
results. Since Fisher (1925) introduced the analysis of variance (ANOVA) 
methods, there has been a preponderance of ANOVA methods and their analogs 
(ANCOVA, MANOVA, AND MANCOVA)- -hereafter referred to as "OVA methods" 
(Thompson, 1985)--used in educational research. From 1969 tc 1978, 
according to Will son (1980), OVA methods were utilized in 56 percent of 
the articles published in the American Educational Research Journal 
(AERJ). Thirty-seven percent of the AERJ articles between 1979-1983 
used OVA techniques (Goodwin & Goodwin, 1985a, 1985b). Reviews by Elmore 
and Woehlke (1988) show that ANOVA and ANCOVA methods make up 25 percent 
of the analytic methods included in publicatii* of three educational 
research journals between 1978-1987. Even thougn the trend painted by 
these reviewers and by others (Gaither & Glorfeld, 1985) show OVA 
utilization on the decline, these methods remain popular (Daniel, 
1989) --although other methods are sometimes more useful (Thompson, 1986). 
Several concerns have been voiced by some researchers regarding this 
continued infatuation with OVA methods. These concerns are beyond the 
scope of this paper, but are explained elsewhere (Cohen, 1968; Thompson, 
1981). 

OVA analytical methods examine the null hypothesis of equality among 
means. If there are more than k = 2 levels in a way or factor and the 
null hypothesis is rejected, the researcher using classical OVA methods 
must then conduct additional tests to determine from where the differences 
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arise (DuRapau, 1988). This is emphasized by Moore (1983, p. 299) when he 
suggests that: 

If we have statistical significance when we have only two 
groups, and thus only two means, we can visually inspect the 
data to determine which group performed better than the 
other. But when we have three or more groups, we need to 
investigate specific mean comparisons. 

Purpose and Scope 

This paper focuses on two methods of performing multiple comparisons 
(post hoc or unplanned versus a priori or planned) for determining exactly 
where statistically significant results arise when there are more than k = 
two groups. The paper summarizes textbook views and recommendations 
regarding the use of these two types of multiple comparisons (Thompson, 
1987). Such a review may be useful in understanding the origins of 
several features of contemporary analytic practice. 

Post Hoc or Unplanned Tests 
When a researcher obtains a significant F in OVA analysis and k = 3 
or more, there is a question about wher le effect is coming from 
(Gravetter & Wallnau, 1085). Huck, Cormier, and Bounds (1974, p. 68) 
remind us that "a researcher cannot stop his analysis after getting a 
significant £." This can be achieved by employing unplanned, post hoc, or 
a posteriori multiple comparisons tests--such as Sheffe, Tukey, or 
Duncan--to determine exactly where statistically significant group 
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differences actually arise within ways (OVA factors) having three or more 
levels (Thompson, 1988a; 1988b). 

A posteriori comparisons may include all possible comparisons and, 
thus, are seen by some researchers as reflecting a lack of pre-experiment 
thought and preparation. Post hoc tests also result in the inflation of 
the experimentwise Type II error rate. Post hoc tests incorporate 
statistical adjustments to correct for inflation of the experimentwise 
Type I error (Fish, 1988) associated with conducting so many tests. This 
causes unplanned tests to have less power in controlling for Type H 
error. 

Huberty and Morris (1988, p. 576) advocate a single contrast test 
procedure and believe that the distinction between planned and post hoc 
tests made by some textbook authors is "drastically overplayed." They 
maintain that "only very few research situations would preclude a 
researcher from specifying all contrasts of interest prior to an 
examination of the outcome measures and/or the outcome 'cell' means. [A 
typical set of contrasts investigated consists of, simply, all pairwise 
comparisons.]" However, this view is somewhat atypical. 

Various methodologists have referred to the use of post hoc testing 
as "milking" (Keppel, 1982, p. 150) or "data snooping" (Kirk, 1969, p. 73; 
1984, p. 360; Pedhazur, 1982, p. 305). In much the same manner. Minium 
and Clarke (1982, p. 321) state: 

Prior to running the experiment, the investigator in our 
example had no well -developed rationale for focusing on a 
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particular comparison between means. His was a "fishing 
expedition"... Such comparisons are known as post hoc 
comparisons, because interest in them is developed "after 
the fact"--it is stimulated by the results obtained, not by 
any prior rationale. 
Kerlinger (1986, p. 219) concedes that a posteriori comparisons can 
be useful in research "for exploring one's data and for getting leads for 
future research," but suggests that a priori comparisons are "perhaps more 
important scientifically." Wise researchers know that all univariate 
statistics are a special case of regression and that OVA methods can be 
utilized using regression approaches to make planned comparisons of 
selected neans. 

Planned or A Priori Tests 
This alternative to post hoc or unplanned tests is achieved by 
constructing planned (also called a priori or focused) comparisons using 
sets of weighting contrasts. It is possible to code the contrasts to test 
for non 'Tend hypotheses focusing on selected groups, or the researcher 
can use tvi-->d contrasts to test hypotheses about the patterns in the means 
across all the groups. Thompson (1987) generated a data set to simulate a 
non-trend contrast analysis in the six-level, one-way case, shown in Table 
1. Contrasts testing for trends in means are provided by Fisher and Yates 
vl957, pp. 90-100) and by Hicks (1973) and could have been used instead of 
non-trend contrasts ;f the interest had been in testing hypotheses about 
patterns in the group means across aVl groups. 
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INSERT TABLE 1 ABOUT HERE 



For each main effect, there is one less orthogonal (i.e., 
uncorrel ated) coding contrast than the number of levels in that way. 
Thus, the total number of coding columns in Table 1 equals the number of 
degrees of freedom, i.e., df = k - 1 where k = the number of levels in the 
way. Each coding column in the example depicts a contrast that represents 
a null hypothesis (DuRapau, 1988, p. 3). 

The information involved in testing omnibus OVA effects is the same 
as that represented in the coding columns as a complete set. The 
difference in the contrast coding lies in the partitioning of the total 
explained sum of squares, SOS explained, of the dependent variable into 
smaller parts using separate one df hypotheses (Thompson, 1985). 

Contrasts do not have to be orthogonal. Winer (1971, p. 175) stated 
that "whether these comparisons are orthogonal or not makes little or no 
difference." Orthogonal planned comparisons have a special appeal, 
however. Keppel (1982, p. 147) stated that, since orthogonal comparisons 
are uncorrelated, decisions regaraing the null hypothesis of one 
comparison (or contrast) is not influenced by decisions regarding any 
other orthogonal comparison. This independence seems to be an important 
factor in the analysis of relationships among means. When each contrast 
(coding column) totals zero and when the sum of the cross-products of each 
pair of contrasts total zero, contrasts are perfectly uncorrelated or 
orthogonal, as are the contrasts presented in Table 1 (Thompson, 1988b). 
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Preference for Planned Comparisons 
For two reasons, planned comparisons are generally better than post 
hoc tests. As stated earlier, the first reason is that planned multiple 
comparisons have greater statistical power against Type II error than 
unplanned multiple comparisons (Thompson, 1985; 1988b; 1988c). Power 
means one is more likely to get statistical significance for fixed sample 
and effect sizes. Thompson (1987, 1988b) demonstrates that with the Table 
1 data. With classical ANOVA there were no statistica'/ly significant 
effects, but significant effects were achieved using planned comparisons 
with regression and planned comparisons. Thus, Kerlinger and Pedhazur 
(1973, p. 131) stated: 

The tests of significance for a priori, or planned, 
comparisons are more powerful than those for post hoc 
comparisons. In other words, it is possible for a 
specific comparison to be not significant when tested by 
post hoc methods but significant when tested by a priori 
methods. 

Similarly, Rosnow and Rosenthal (1989, p. 1281) deplore the 
"overreliance on omnibus tests of diffuse hypotheses that although 
providing protection for some investigators from the dangers of 'data 
mining' with multiple tests performed as if each were the only one 
considered" because omnibus tests generally do not: 

tell us anything we really want to know. As Abel son (1962) 
pointed out long ago in the case of analysis of variance 
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(ANOVA), the problem is that when the null hypothesis 
is accepted, it is frequently because of the insensitive 
omnibus character of the standard F-test as much as by 
reason of sizable error variance. All thp while that a 
particular predicted pattern among the means is evident to 
the naked eye the standard F-test is often insufficiently 
illuminating to reject the null hypothesis that several 
means are statistically identical. 
There are critics of significance testing (Carver, 1978) who would 
agree with Thompson (1988a, p. 100) that "significance is not, however, 
the end-all and be-all of research." But there can be no doubt that 
significance is paramount in research for graduate students: 

It may not be an exaggeration to say that for many PhD 
students, for whom the .05 alpha has acquired almost an 
ontological mystique, it can mean joy, a doctoral degree, 
and a tenure-ir«ck position at a major university if their 
dissertation p is less than ,05. However, if the p is 
greater than .05, it can mean ruin, despair, and their 
advisor's suddenly thinking of a new control condition that 
should be run... We only wish to emphasize that dichotomous 
significance testing has no otological basis. That is, we 
want to underscore that, surely, God loves the .06 nearly as 
much as the .05, Can there be any doubt that God views the 
strength of evidence for or against the null as a fairly 
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continuous function of the magnitude of p (Rosnow & 
Rosenthal, 1989, p. 1277). 

The second and more substantial reason for choosing to employ a 
priori contrasts is because planned comparisons tend to make the 
researcher think --planned comparisons must be drafted prior to data 
collection and generally only a given number are stated in a study. 
Usually this is helpful. Keppel (1982, p. 165) spoke of planned 
comparisons as "the motivating force behind an experiment" and stated that 
planned comparisons "represent an interest in particular combinations of 
condit1ons--not in the overall experiment." Diligently planning the 
design of the research effort is the foundation for planned comparisons. 
This is in opposition to post hoc methods that may lead to haphazard 
research design that waits to focus on particular comparisons between 
means until achieving a significant omnibus overall F test. 

Researchers who feel ANOVA is the best analytic method for their 
study might get more interpretable, and more correct results by using 
ANOVA through regression methods with planned comparisons when there are 
more than two levels in any one way. Researchers might also take notice 
that there is NEVER a research in which the regression statistical 
procedure is nit at least as good as OVA methods. There is nothing that 
can be accomplished by OVA methods that cannot be done by regression 
approaches. An added perk in using regression analyses for data higher 
than nominal scale is that variance is not squandered. "Amrng the 
practical advantages of contrasts," according to Rosnow & Rosenthal, 1989, 
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p. 1281), "are that they can be easily computed with a pocket calculator, 
can be computed on the data in published reports as well as with original 
data, and most important, usually result in increased power and greater 
clarity of substantive interpretation." 

Analysis of Table 1 

Table 1 data can by analyzed to concretely illustrate the points made 
thus far. These contrasts are orthogonal because the five coding columns 
(k - 1 = 5) each total zero and the sums of their cross-products also all 
equal zero. There is no interaction in this one-way, balanced design, 
hence no interaction coding columns are developed for these data. 

Classical omnibus analysis of variance methods were employed to 
determine if any differences among the means of the groups existed for the 
data in Table 1. As shown in Table 2, the omnibus test of differences 
among the six group means is not statistically significant (F = 1.5, df = 
5/6, £ = .3155). Additionally, if a post hoc test was run in violation of 
traditional practice (because the one-way ANOVA did not involve a 
significant F), there still would not be any statistically significant 
differences among means for these data. 



INSERT TABLE 2 ABOUT HERE 



ANOVA through regression is more appropriate with this study because 
there are six levels to this one-way design and k > 2. As illustrated in 
Table 3, using the regression approach to ANOVA with planned comparisons, 

ERIC 



10 

the mean of the two level -six subjects versus the mean of the remaining 10 
subjects is statistically significant (£ = 12.5, .df = 1/6, = .0054). 
The null hypothesis, i.e., that the two scnool board r.embers mean 
attitude-toward-school score (DV) equals the mean for the other 10 
subjects, is rejected (DuRapau, 1988; Thompson, 1987). 



INSERT TABLE 3 ABOUT HERE 

All the same conclusions apply in the multivariate case, as 
emphasized by Swaminathan (1989, p. 231): 

The often advocated procedure of following up the rejection 
of the null hypothesis with a more powerful multiple 
comparison procedure should be discouraged. First, the 
overall rejection of the null hypothesis does not guarantee 
that any meaningful contrast amcng the means will be 
significant. . .Second, since the overall test and the 
follow up procedure are unrelated, significant contrasts 
may be found even when the null hypothesis would not have 
been rejected. Third, follow up multiple comparison 
proceaures which are unrelated to the overall test result in 
an inflation of the experiment-wise error rate. If multiple 
comparisons are of primary interest, a suitable multiple 
comparison procedure can be used without first 
performing an overall test. 
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Review and Summarization of Textbook Views 
According to research by Wlllson (1982), from 1950 until the 1970's 
the research design statistics instruction for education graduate students 
stressed analysis of variance (ANOVA) techniques based on textbooks by 
Lindquist (1953), Winer (1962, 1971), Kirk (1969), Glass and Stanley 
(1970) and others. Not until the appearance of texts by Ward and Jennings 
(1973) and Kerlinger and Pedhazur (1973) could much emphasis on regression 
approaches be found. Willson (1982) attributes to these later texts the 
increased use of regression statistical approaches in OVA situations, as 
indicated by a review (Willson, 1980) of 10 years' research in the 
American Educational Research Journal . Only limited use of regression 
approaches was recorded from 1969 to 1978. In 1978-1979 regression 
methods were used extensively, eccording to Willson (1982). 

An evaluation of five organizational behavior research journals from 
1976 to 1985 was conducted by Gaither end Glorfeld (1985) with 
regression/correlation statistical analysis method being the most 
frequently employed technique. This shows the same trends in 
organizational behavior journals that is being documented in education 
journals, i.e., an increased use of regression methods. 

A compendium of textbook views on planned versus post hoc tests was 
compiled from selected books to determine textbook views and 
recommendations regarding the use of planned versus post hoc multiple 
comparisons of means. Results of this compilation, along with selected 
quotations, are presented in Table 4. 
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INSERT TABLE 4 ABOUT HERE 

Each book in Table 4 was randomly pulled and checked to see if 
planned or post hoc comparisons was indexed. Alternately, a reference for 
multiple comparisons of means was sought. From 1960-1967 no references 
were indexed for planned or post hoc comparisons (Armore, 1966; Bloomers & 
Lindquist, 1960; Bryant, 1960; Chase, 1967). Although Clark and Schkade 
(1969) do not cover planned versus post hoc comparisons, Kirk does in his 
19G9 book and explains that "techniques that have been developed for data 
snooping. . .are referred to as a posteriori or post-hoc tests" (p. 73). 
Glass and Stanley (1970) and Winer (1971), however, cover only post hoc 
comparisons in their texts. These findings corroborate Willson's (1980, 
1982) view that statistics instruction did not historically offer the 
alternative planned comparison procedure. 

Keppe'j (1973) took the stand that "post-hoc comparisons may be 
described as data 'sifting' (p. 93), intensive 'milking' (p. 133), and 
systematic 'fishing' (p. 136)." He stated that "it is clear, then, that 
tests of planned comparisons are a desirable alternative to the omnibus F 
test" (p. 93). This corresponds to Willson's findings (1982) that 1973 
texts represented the beginning of emphasis on regression approaches 
(Kerlinger & Pedhazur, 1973; Ward ?i Jennings, 1973) and might further 
substantiate Willson's claim that these texts presenting plannpi 
comparisons helped contribute to the increased use of regression 
approaches in journal articles. Contrary to Willson's study, the present 
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research was not limited to textbooks used expressly in education graduatw 
programs, but rather was a sampling across disciplines. 

The use of planned versus post hoc comparisons continues to be a much 
debated issue with proponents of each view taking strongly felt and 
heatedly argued positions. But little or no emphasis was given this topic 
in many textbooks of the 1970' s (Bhattacharyya & Johnson, 1977; Chase, 
1976; Chiswick & Chiswick, 1975; Clark & Schkade, 1974; Glass & Stanley, 
1970; Huck, Cormier, & Bounds, 1974; Jacobson, 1976; Mitroff & Kilmann, 
1978; Nachmias & Nachmias, 1976; Winer, 1971). Even though Cohen and 
Cohen (1975) c.-^lled "planned comparisons. . .the most elegant multiple 
comparison proced ire [with] good power characteristics," they cautioned it 
could "only infrequently be employed in behavioral science investigations 
because the questions to be put to the data are simply not usually 
independent" (p. 158). Rosnow and Rosenthal (1989, p. 1282) credit 
current statistical textbooks with describing "the logic and the machinery 
of contrast analysis," but recognize that "one still sees contrasts 
employed all too rarely. That is a real pity given the precision of 
thought and theory they encourage and (especially relevant to these times 
of publication pressure) given the boost in power conferred with the 
resulting increase in .05 asterisks." 

Statistical analysis is fast becoming a computerized automated 
process, allowinr; for faster, more accurate analyses of data. Statistical 
program documentation such as Norusis (1988a, 1988b), however, offers 
little conceptual framework for the student to become acquainted with 
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planned versus post hoc tests. This debate is one of substance and must 
not be overlooked in the classroom, or the researchers of tomorrow will 
lack important knowledge to design sound studies. 

Recommendations 

The present paper calls for increased emphasis on planned versus post 
hoc tests in the graduate student's statistics program. In addition, 
there is a need fo»^ further investigation of current statistics textbooks 
being used in the different schools of education, business, and the 
behavioral sciences to determine if the issue of planned versus post hoc 
tests is being adequately addressed in instructional materials. 

A final note, echoing Rosnow and Rosenthal (1989, p. 1282), is that: 
"much of what we have said has been said before, but it is important that 
our graduate students hear it all again so that the next generation of 
psychological scientists is aware of tlie existence of these pitfalls and 
of the ways around them." 
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Table 1. Hypothetical Data for Attitudes Toward School Study {n=12) 



ID 


GROUP 


LEVEL DV 


CI 


C2 


C3 


C4 


Co 


1 


Students 


1 


10 


■ 1 


-1 


:} 


-1 


:} 


2 






20 


-i 


-1 




-1 




3 


Teacher Aides 


2 


10 


1 


-1 




-1 




4 






20 


1 


-1 




-1 




5 


Teachers 


3 


10 


0 


2 




-1 




6 






20 


0 


2 




-1 




7 


Principals 


4 


10 


0 


0 


3 


-1 




8 




20 


0 


0 


3 


-1 




9 


Superintendents 


5 


10 


0 


0 


0 


4 




10 




20 


0 


0 


0 


4 




11 


Board Members 


6 


25 


0 


0 


0 


0 


5 


12 






35 


0 


0 


0 


0 


5 




Table 


2. 


One-way ANOVA Results 






SOURCE 


SOS df 


MEAN SQUARE 




F 


p 


ETA SQUARE 


Between 375.000 5 




75.000 


1.500 


.3155 




.55556 


Error 


300.000 6 




50.000 












Total 


675.000 11 

















NOTE: If, in violation of conventional practice, post hoc tests 
were conducted in the absence of a significant omnibus result, the 
tests would still not identify any two groups with means 
significantly different at alpha equal 0.05. 



Table 3. Planned Comparison Results 



CONTRAST 






MEAN 






ETA 


SOURCE 


SOS 


df 


SQUARE 


F 


P 


SQUARE 


CI 


.000 


1 


.000 


0.000 




.0000 


C2 


.000 


1 


.00 


0.000 




.0000 


C3 


.000 


1 


.Of 


0.000 




.0000 


C4 


.000 


1 


J 


0.000 




.0000 


C5 


375.000 


1 


375.000 


12.500 


.0054 


.5556 


Error 


300.000 


6 


50.000 








Total 


675.000 


11 











SOURCE: Tables 1, 2, and 3 from "The Importance of a priori 
Contrasts in Analysis of Variance Research," Thompson (1987). 
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Table 4. Textbook Views on Planned Versus Post Hoc Tests 



CONTAINS CONTAINS 



QUOTATIONS 



DATE 


AUTHOR 


PLANNED 


POST HOC 


1960 


BLOOMERS & LINDQUIST 


NO 


NO 


1960 


BRYANT 


NO 


Kin 
NU 


1966 


ARMORE 


NO 


NO 


1967 


CHASE 


NO 


NO 


1969 


CLARK h SCHKADE 


NO 


NO 


1969 


KIRK 


YES 


YES 


1970 


GLASS & STANLEY 


NO 


YES 


1971 


WINER 


NO 


YES 


1973 


KEPPEL 


YES 


YES 



i.chniques that have been developed for data 
Snooping.. .are referred to as a posteriori or 
post-hoc tests, (p. 73) 

Most multiple comparison procedures are designed 
to be used after the null hypothesis of no 
treatment differences in an ANOVA has been 
rejected, (p. 383) 

Tu',ey and Scheffe developed methods for 
constructing simultaneous confidence intervals 
which avoid the pitfall of permitting the Typs I 
error to become excessively large, (p. 175) 

...a significant F allows* if not demands, a 
further analysis of the data. (p. 87) 
...It is clear, then, that tests of planned 
comparisons are a desirable alternative to the 
omnibus F test. (p. 93) 

...post-hoc comparisons may be described as 'data 
sifting,*' where an experimenter is sorting 
through a large number of comparisons in the 
hope of finding something significant, (p. 93) 
...post-hoc comparisons often take the form of an 
intensive "milking" of a set of results, (p- 133) 
...This systematic "fishing" from the "pool" of 



1974 CLARK & LAWRENCE 

1974 HUCK, CORMIER, & 
BOUNDS 



1975 CHISWICK & CHISWICK 



NO 
NO 

NO 



NO 

YES 

NO 



1975 COHEN & COHEN 



YES 



YES 



1976 CHASE 

1976 JACOBSON 

1976 NACHMIAS & NACHMIAS 

1977 BHATTACHARYYA & 
JOHNSON 

1978 MITROFF & KILMANN 



YES 

NO 
NO 
NO 
NO 



NO 
NO 
NO 
NO 



all possible pairwise comparisons greatly 
increases the number of comparisons we are 
effectively considering and, therefore, the 
probability that we will make more Type I 
errors, (p. 136) 



...a researcher cannot stop his analysis after 
getting a significant F; he must locate the 
cause of the significant F. (p. 68) 

...regression analysis now tends to be preferred 
to ANOVA, except when tests for "any interaction" 
between two explanatory variables are of primary 
importance, (p. 211) 

Planned comparisons are generally considered the 
most elegant multiple comparison procedure and 
have good power characteristics, but... can only 
infrequently be employed in behavioral science 
investigations because the questions to be put to 
the data are simply not usually independent... 
(p. 158) 

..."data snooping" is an important part of the 
research process, out unless Type I error is 
controlled.. .the experimentwis:e rate of spuriously 
significant t values on comparisons becomes 
unacceptably high. (p. 158) 



Such "data snooping" is 
comparisons. .. (p. 188) 



known as post hoc 
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...there are many other ways in which mutually 
orthogonal X vectors can be constructed. . .some 
sets may be more meaningful than others in a 
given experiment, (p. 85) 



...it is possible that a particular planned 
comparison will be significant in the absence of 
an overall significant F from the ANOVA. (p. 386) 

In most situations, you want to pinpoint exactly 
where the differences are. To do this, you must 
use multiple comparison procedures. .. (pp. 262-263) 

A variety of special techniques, termed multiple 
comparison procedures, are available for 
determining which population means are different 
from each other... (p. 119) 

...planned comparisons are more powerful than the 
data snooping permitted by post hoc comparisons, 
(p. 364) 
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APPENDIX A 
Selected SPSS-X Control Cards 



TITLE '*****OMNIBUS no POSTHOC no A PRIORI yes' 

FILE HANDLE BT/NAME='APRIORI .DTA' 

DATA LIST FILE=BT/LEV 1 DV 2-4 

COMPUTE C1=0 

COMPUTE C2=0 

COMPUTE C3=0 

COMPUTE C4=0 

COMPUTE C5=0 

IF (LEV EQ 1)C1=-1 

IF (LEV EQ 2)C1=1 

IF (LEV EQ 3)C2=2 

IF (LEV EQ 1 OR LEV EQ 2)C2=-1 

IF (LEV EQ 4)C3=3 

IF (LEV LT 4)C3=-1 

IF (LEV EQ 5)C4=4 

IF (LEV LE 4)C4=-1 

IF (LEV EQ 6)C5=5 

IF (C5 EQ 0)C5=-1 

REGRESSION VARIABLES=DV CI TO C5/DESCRIPTIVES=ALL/ 
CRITERIA=PIN(.95) P0UT(.999) TOLERANCE( .00001)/DEPENDENT=DV/ 
FNTER C5/ENTER C4/ENTER C3/ENTER C2/ENTER CI/ 



Note: Adapted from "The importance of a priori contrasts in analysis of 
variance research: (Thompson, 1987). 
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